Method and apparatus for recognizing touch gesture

ABSTRACT

A method and an apparatus for recognizing a touch gesture are disclosed, in which the apparatus may obtain a depth image in which a touch object and a background area are captured, detect a touch input applied by the touch object to the background area in a touch detection area, and recognize a touch gesture associated with the touch input by tracking a change in the touch input.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit under 35 USC 119(a) of Korean PatentApplication No. 10-2014-0166824, filed on Nov. 26, 2014, in the KoreanIntellectual Property Office, the entire disclosure of which isincorporated herein by reference for all purposes.

BACKGROUND

1. Field

The following description relates to technology for recognizing a touchgesture.

2. Description of Related Art

Recently, various user interfaces are being developed to provide a userwith a convenient control environment. Among these interfaces, atouchless control type may control a terminal without a contact with ora touch on a touch screen or a button of the terminal. Such a touchlesscontrol type may apply, for example, a method using sensing informationtransmitted from various proximity sensors, a method using a temperatureof a body portion of a user, and a method using an image in which afinger is captured.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In one general aspect, there is provided a method of recognizing a touchgesture, the method including obtaining a depth image in which a touchobject and a background area are captured, detecting, from the depthimage, a touch input applied by the touch object to the background areain a touch detection area, and recognizing a touch gesture associatedwith the touch input by tracking a change in the touch input.

The method may further include learning the background area, anddetermining the touch detection area from the learned background area.The detecting of the touch input may be performed in the touch detectionarea based on the learned background area.

The detecting of the touch input may include determining a touchdetection area to detect the touch input based on the learned backgroundarea, and detecting the touch input within the touch detection area.

The touch detection area may include a portion of the learned backgroundarea and an adjacent area of a boundary of the learned background area.

The learning of the background area may include determining whether aninitial background area on an initial depth image satisfies apredetermined learning condition and learning the background area basedon the initial depth image when the background satisfies thepredetermined learning condition.

The method may include: determining whether a predetermined learningcycle arrives; obtaining a learning depth image when the predeterminedlearning cycle arrives; determining, when the predetermined learningcycle arrives, whether a touch event is present in a touch detectionarea of the learning depth image based on learning a background area forlearning included in the learning depth image; extracting a an adjacentarea that is adjacent to a boundary of the background area for learningfrom the learning depth image when the touch event is absent in thetouch detection area, and learning a new touch detection area based onthe extracted adjacent area.

The detecting of the touch input may include determining whether thetouch object makes contact on the touch detection area, and detecting achange in the touch input when the touch object is in contact on thetouch detection area.

The determining of whether the touch object makes contact on the touchdetection area may include determining whether the touch object makescontact with a boundary of the learned background area on the touchdetection area based on a shape of the learned background area.

The detecting of the touch input may include determining at least one ofa location of the touch input and a touch angle of the touch input. Thetracking of the change in the touch input may include tracking at leastone of a change in the location of the touch input and a change in thetouch angle of the touch input.

The detecting of the touch input may include detecting touch inputswithin the touch detection area. The recognizing of the touch gesturemay include recognizing a touch gesture associated with the touch inputsbased on a mutual relation between the detected touch inputs.

According to another general aspect, a computer-readable storage mediummay comprise a program comprising instructions to cause a computer toperform the method of recognizing a touch gesture.

In another general aspect, there is provided a method of recognizing atouch gesture, the method including learning, based on a depth image, abackground area to be in contact with a touch object, determining atouch detection area based on the learned background area, detecting atouch input by the touch object based on a touch object area in thetouch detection area, and recognizing a touch gesture corresponding to amotion of the touch input.

The learning of the background area may include determining whether apredetermined learning cycle arrives, determining, when thepredetermined learning cycle arrives, whether a touch event is presentin the depth image, extracting a background area for learning from alearning depth image when the touch event is absent in the depth image,and learning the background area to be in contact with the touch objectbased on the extracted background area.

The detecting of the touch input may include determining whether thetouch object makes contact within the touch detection area, anddetecting a change in the touch input when the touch object is incontact with the touch detection area.

The detecting of the touch input may include determining at least one ofa location of the touch input and a touch angle of the touch input. Therecognizing of the touch gesture may include recognizing a touch gestureassociated with the touch input by tracking at least one of a change inthe location of the touch input and a change in the touch angle of thetouch input.

In still another general aspect, there is provided an apparatus forrecognizing a touch gesture, the apparatus including a depth sensorconfigured to obtain a depth image in which a touch object is captured,and a processor configured to detect, from the depth image, a touchinput applied by the touch object in a touch detection area andrecognize a touch gesture associated with the touch input by tracking achange in the touch input.

The apparatus may further include a learner configured to learn abackground area for an adjacent area of that is adjacent to a boundaryof a background area detected in the depth image. The processor may beconfigured to update the touch detection area based on updating thelearned background area at predetermined learning cycles.

The learner may determine whether a touch event is present on the touchdetection area when a predetermined learning cycle arrives, and toextract a learning background area from a learning depth image and learnthe background area from the learning depth image when the touch eventis absent in the touch detection area.

In a further general aspect, there is provided an apparatus forrecognizing a touch gesture, the apparatus including a depth sensorconfigured to obtain a depth image, a learner configured to learn abackground area with which a touch object makes contact from the depthimage obtained by the depth sensor, and a processor configured to detecta touch input applied by the touch object from the depth image in whichthe touch object is captured and recognize a touch gesture correspondingto a motion of the touch input.

The processor may be configured to determine a touch detection area fordetecting the touch input. The touch detection area may include aportion of the learned background area and an area adjacent to aboundary of the learned background area.

The touch detection area may be only a portion of an entire area of thedepth image.

The processor may be configured to update the touch detection area basedon updating the learned background area at predetermined learningcycles.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A through 1C illustrate examples of implementation of anapparatus for recognizing a touch gesture.

FIG. 2 is a flowchart illustrating an example method of recognizing atouch gesture.

FIG. 3 is a flowchart illustrating an example method of learning abackground area.

FIG. 4 is a diagram illustrating an example operation of learning abackground area.

FIG. 5 is a flowchart illustrating another example method of learning abackground area.

FIG. 6 is a diagram illustrating another example operation of learning abackground area.

FIG. 7 is a flowchart illustrating an example method of detecting atouch input from a depth image.

FIG. 8 is a diagram illustrating an example operation of detecting atouch input from a depth image.

FIG. 9 is a flowchart illustrating an example method of analyzing atouch input.

FIG. 10 is a diagram illustrating an example operation of analyzing atouch input.

FIG. 11 is a flowchart illustrating another example method ofrecognizing a touch gesture.

FIG. 12 is a diagram illustrating an example configuration of anapparatus for recognizing a touch gesture.

FIGS. 13A and 13B illustrate other examples of implementation of anapparatus for recognizing a touch gesture.

Throughout the drawings and the detailed description, unless otherwisedescribed or provided, the same drawing reference numerals will beunderstood to refer to the same elements, features, and structures. Thedrawings may not be to scale, and the relative size, proportions, anddepiction of elements in the drawings may be exaggerated for clarity,illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, apparatuses,and/or systems described herein. However, various changes,modifications, and equivalents of the systems, apparatuses and/ormethods described herein will be apparent to one of ordinary skill inthe art. The progression of processing steps and/or operations describedis an example; however, the sequence of and/or operations is not limitedto that set forth herein and may be changed as is known in the art, withthe exception of steps and/or operations necessarily occurring in acertain order. Also, descriptions of functions and constructions thatare well known to one of ordinary skill in the art may be omitted forincreased clarity and conciseness.

The features described herein may be embodied in different forms, andare not to be construed as being limited to the examples describedherein. Rather, the examples described herein have been provided so thatthis disclosure will be thorough and complete, and will convey the fullscope of the disclosure to one of ordinary skill in the art.

FIGS. 1A through 1C illustrate examples of implementation of anapparatus for recognizing a touch gesture.

Referring to FIGS. 1A through 1C, the apparatus for recognizing a touchgesture, hereinafter also referred to as a touch gesture recognizingapparatus, recognizes a touch gesture made by a motion of a touchobject. For example, a user may input a user input to a dorsum or a palmof a hand or an arm area of the user using the touch object. The touchobject may be, for example, a finger, a pen or a stylus. The touchgesture recognizing apparatus recognizes the user input using a depthsensor and recognizes a gesture pattern intended by the user. Forexample, the user may input the user input, for example, an oriented orcircular symbol and a character, to the dorsum of the hand.

The touch gesture recognizing apparatus obtains depth information on thetouch object using a depth sensor and recognize the touch gesture basedon the obtained depth information. The depth sensor may be, for example,a camera. The depth information on the touch object may includeinformation on a distance between the touch object and the touch gesturerecognizing apparatus including the depth sensor.

The touch gesture recognizing apparatus generates, based on a depththreshold value, a depth image in which an area deviating from an angleof view of the depth sensor is obscured. In the depth image, an areapresent in a distance exceeding the depth threshold value may be set asa depth value. For example, in the depth image, pixels in the areapresent in the distance exceeding the depth threshold value may be setas “0” or “maximum” depth value, for example, 255, based on a type ofthe depth sensor. When a background area is the dorsum of the hand ofthe user, a remaining area from which the dorsum is excluded in thedepth image may be set as 0 or the maximum depth value.

The touch gesture recognizing apparatus may obtain a stream of the depthimage in which the touch object is captured, and recognize the userinput by tracking the motion of the touch input made by the touch objectbased on the stream of the depth image. The touch gesture recognizingapparatus may identify the touch input made by the touch object based onthe depth information on the touch object that may be changed over time,and recognize the touch gesture intended by the user by analyzing theidentified touch input.

The touch gesture recognizing apparatus may perform gesture recognitionthat is robust against a movement of the dorsum by continuously learninga background area included in the depth image. The background area mayinclude, for example, the dorsum or the palm of the hand, and the armarea. The touch gesture recognizing apparatus may determine whether asingle object or multiple touch objects appearing in the depth imagemake contact with the background area, and perform the tracking byconverting locations of the touch objects in contact with the backgroundarea to two-dimensional (2D) data. The touch gesture recognizingapparatus may recognize the gesture pattern intended through the userinput by the user based on a result of the tracking, and generate acontrol signal corresponding to the recognized gesture pattern.

The touch gesture recognizing apparatus may be included in variouswearable devices to operate. For example, as illustrated in the examplesof FIGS. 1A through 1C, a touch gesture recognizing apparatus 110 may beincluded in a wearable device, such as a smart watch, worn around awrist to operate. The touch gesture recognizing apparatus 110 may obtaina depth image in which a touch object 140 is captured using a depthsensor 120, and recognize a gesture pattern made by a motion of thetouch object 140 on a dorsum 130 of a hand by analyzing a change in thedepth image over time. In the examples of FIGS. 1A through 1C, brokenlines commencing from the depth sensor 120 may form an angle of view,which indicates a sensible range that may be sensed by the depth sensor120.

The touch gesture recognizing apparatus 110 tracks the motion of thetouch object 140 by analyzing the change in the depth image over time,and recognizes the gesture pattern corresponding to the motion of thetouch object 140. For example, the touch object 140 may refer to afinger of the user or an object in a form of a pen or a stylus. Thetouch gesture recognizing apparatus 110 generates information on therecognized gesture pattern, and the wearable device performs aninterface operation corresponding to the recognized gesture pattern.

For example, referring to FIG. 1A, the touch gesture recognizingapparatus 110 tracks the motion of the touch object 140 on the dorsum130 based on the depth image obtained by the depth sensor 120, and thetracked motion of the touch object 140 is displayed through the wearabledevice.

Referring to FIG. 1B, the touch gesture recognizing apparatus 110 tracksa change in a touch angle in a state in which the touch object 140 is incontact with the dorsum 130 based on the depth image obtained by thedepth sensor 120, and the wearable device may perform a controloperation corresponding to the tracked change in the touch angle of thetouch object 140. In the example of FIG. 1B, the user may experience aneffect that may be experienced through a joystick interface by adjustingthe touch angle of the touch object 140.

Referring to FIG. 1C, the touch gesture recognizing apparatus 110 maytrack motions of multiple touch objects 140 based on a depth imageobtained by the depth sensor 120, and the wearable device may perform acontrol operation corresponding to the motions of the touch objects 140.When the user makes a spreading gesture by increasing a distance betweentwo fingers, which are the touch objects 140, while the two fingers arein contact with the dorsum 130, a control operation through whichcontents displayed on the wearable device are enlarged may be performed.

FIG. 2 is a flowchart illustrating an example method of recognizing atouch gesture. The method of recognizing a touch gesture may beperformed by a touch gesture recognizing apparatus.

Referring to FIG. 2, in operation 210, the touch gesture recognizingapparatus learns a background area included in a depth image. Thebackground area refers to an area to be in contact with a touch object.The touch gesture recognizing apparatus may learn a background areacorresponding to a dorsum or a palm of a hand or an arm area of a userto which a touch input is input by the touch object. The touch gesturerecognizing apparatus may learn a depth image of a background area in anabsence of the touch object in order to distinguish between thebackground area in the depth image and a touch object area in which thetouch object appears in the depth image.

The touch gesture recognizing apparatus may determine a background areato be used for the touch input by learning a background area included inan initial depth image obtained at an initial state of an operation. Thetouch gesture recognizing apparatus may extract the background area fromthe initial depth image, and learn a background area to be in contactwith the touch object based on the extracted background area. A detailedoperation of learning the background area by the touch gesturerecognizing apparatus at the initial state will be described later withreference to FIGS. 3 and 4.

Subsequent to the initial state, the touch gesture recognizing apparatusmay continuously learn a background area included in the depth image.For example, every time a predetermined learning cycle arrives, thetouch gesture recognizing apparatus may learn a background area from alearning depth image, and update the background area to be in contactwith the touch object based on a result of the learning. The touchgesture recognizing apparatus may determine whether a touch event ispresent in the touch detection area based on learning a background area.When the touch event is absent from the learning depth image, the touchgesture recognizing apparatus may extract the background area from thelearning depth image, and learn the background area to be in contact onthe touch object based on the extracted background area. The touch eventmay refer to an event indicating that the touch object makes contactwith the background area. A detailed operation of continuously learningthe background area by the touch gesture recognizing apparatussubsequent to the initial state will be described later with referenceto FIGS. 5 and 6.

The touch gesture recognizing apparatus may determine a touch detectionarea based on the learned background area. The touch detection area mayinclude a portion of the learned background area and an adjacent areathat is adjacent to a boundary of the learned background area. Forexample, when the background area is the dorsum of the hand of the user,the touch detection area may include a portion of the dorsum and anadjacent area that is adjacent to a boundary of the dorsum.

Searching for a touch input made by the touch object from all areasincluded in the depth image and recognizing the found touch input may beineffective. The touch gesture recognizing apparatus may determine aportion of the depth image to be the touch detection area for detectingthe touch input and thus, effectively detect the touch input from thedepth image. The touch gesture recognizing apparatus may reduce anamount of computation by verifying whether the touch input is presentbased on the touch detection area, which is a portion of an entire areaof the depth image.

In operation 220, the touch gesture recognizing apparatus determineswhether the touch object makes contact with the background area based onthe depth image in which the touch object is captured based on the touchdetection area. The depth image may include information on a depth fromthe touch gesture recognizing apparatus to the touch object. The touchgesture recognizing apparatus may determine whether the touch objectmakes contact with the background area within the touch detection area.For example, the touch gesture recognizing apparatus may determinewhether the touch object makes contact with the background area based ona blob-shaped touch input area included in the touch detection area.

In operation 230, the touch gesture recognizing apparatus detects achange in the touch input made by the touch object. The touch gesturerecognizing apparatus detects a motion of the touch input made by thetouch object based on a touch object area included in the touchdetection area. The touch gesture recognizing apparatus may determine alocation of the touch input or a touch angle of the touch input.

The touch gesture recognizing apparatus may calculate an x coordinatevalue on spatial coordinates of the touch object area in a state inwhich the touch object is in contact with the background area. Forexample, the touch gesture recognizing apparatus may determine a pointat a center of an area within edges of the touch object area to be the xcoordinate value of the touch input based on a width direction of thedepth image. The touch gesture recognizing apparatus may set, within thetouch object area, an identification area corresponding to an edge ofthe touch object, and determine a depth value of the set identificationarea to be a z coordinate value on the spatial coordinates of the touchinput. The touch gesture recognizing apparatus may convert the locationof the touch input to 2D coordinate data of an x axis and a z axis basedon the x coordinate value and the z coordinate value of the touch input.The touch gesture recognizing apparatus may determine the touch angle ofthe touch input by additionally calculating tilts in an x axis directionand a z axis direction.

A detailed operation of detecting the touch input from the depth imageand detecting the change in the touch input by the touch gesturerecognizing apparatus will be described later with reference to FIGS. 7through 10.

In operation 240, the touch gesture recognizing apparatus recognizes atouch gesture corresponding to the motion of the touch input. The touchgesture recognizing apparatus identifies at least one touch object incontact with the background area, and analyze the motion of the touchinput made by the touch object by tracking a moving route of theidentified touch object. The touch gesture recognizing apparatusdetermines a gesture pattern intended by the user by analyzing a changein the 2D coordinate data of the touch input.

For example, the touch gesture recognizing apparatus may recognize thetouch gesture associated with the touch input by tracking a change inthe location of the touch input or a change in the touch angle of thetouch input. The touch gesture recognizing apparatus may determine thegesture pattern intended by the user through the touch object by parsingthe change in the location of the touch input or the change in the touchangle of the touch input, which are indicated in the depth image overtime. The touch gesture recognizing apparatus may then generateinformation on the determined gesture pattern. The touch gesturerecognizing apparatus may determine a motion pattern intended by theuser using a learning based method or an existing gesture recognizingmethod. The information on the determined gesture pattern may betransmitted to another device, and the device may perform an interfaceoperation corresponding to the gesture pattern.

For example, characters or gesture icons corresponding to the motion ofthe touch input may be displayed through another device. When the motionpattern corresponding to the motion of the touch input is a character,the device may display the character using an optical characterrecognition (OCR) method and a spelling corrector. When the motionpattern corresponding to the motion of the touch input indicates agesture icon, the device may display the gesture icon on a screen. Whena plurality of touch inputs is detected, the touch gesture recognizingapparatus may determine an interface operation based on a mutualrelation between motions of the touch inputs.

FIG. 3 is a flowchart illustrating an example method of learning abackground area. According to the method of FIG. 3, a touch gesturerecognizing apparatus may learn a background area from a depth image todistinguish between the background area corresponding to a contactsurface and a touch object area of a touch object.

Referring to FIG. 3, in operation 310, the touch gesture recognizingapparatus obtains an initial depth image using a depth sensor in aninitial state of an operation.

In operation 320, the touch gesture recognizing apparatus extracts acandidate background area from the initial depth image. For example, thetouch gesture recognizing apparatus may extract a candidate backgroundarea corresponding to a dorsum of a hand of a user from the initialdepth image using an existing connected component image processingmethod. In the connected component method, the initial depth image maybe scanned and the pixels of the initial depth image may be grouped intocomponents based on pixel connectivity such that all pixels in aconnected component share similar pixel intensity values and are in someway connected with each other. Once all of the pixels have been grouped,each pixel may be labeled with a gray level or a color according to thecomponent to which it was assigned. The touch gesture recognizingapparatus may extract the candidate background area in which noise andother objects are eliminated from the depth image through the connectedcomponent method.

In operation 330, the touch gesture recognizing apparatus calculates acurvature of the candidate background area. For example, the touchgesture recognizing apparatus may calculate a boundary curvature of thecandidate background area.

In operation 340, the touch gesture recognizing apparatus determineswhether the curvature of the candidate background area satisfies apredetermined learning condition. The touch gesture recognizingapparatus may determine whether the candidate background area includesan area indicating the touch object or another object based on thecurvature of the candidate background area. For example, when thecandidate background area includes the area indicating the touch objector another object instead of including only the dorsum of the hand, asection may exist in which a curvature value of the candidate backgroundarea is suddenly and significantly changed. When such a section exists,the touch gesture recognizing apparatus may determine that the candidatebackground area does not include only the dorsum of the hand and thecurvature of the candidate background area does not satisfy thepredetermined learning condition.

In operation 350, when the curvature of the candidate background areadoes not satisfy the predetermined learning condition, the touch gesturerecognizing apparatus outputs a “touch off” message. The touch offmessage may refer to a message requiring the user to touch off, or movethe touch object out of contact with, the background area. Subsequent tothe output of the touch off message, the touch gesture recognizingapparatus may obtain an initial image again to learn a background area.

Also, when an area estimated not to be a background area in an upperportion of the initial depth image or an area longitudinally traversingthe initial depth image exists, the touch gesture recognizing apparatusmay determine that the candidate background area does not include only abackground area, and may learn a background area by obtaining an initialdepth image again.

In operation 360, when the curvature of the candidate background areasatisfies the predetermined learning condition, the touch gesturerecognizing apparatus learns the background area based on the candidatebackground area extracted from the initial depth image. The touchgesture recognizing apparatus may perform the learning to recognize thecandidate background area extracted from the initial depth image to be abackground area for detecting a touch input.

FIG. 4 is a diagram illustrating an example operation of learning abackground area.

Referring to FIG. 4, a depth image 410 indicates an initial depth imageobtained by a depth sensor disposed on a side of a watch type wearabledevice embedded with a touch gesture recognizing apparatus. The depthimage 410 may be obtained when the depth sensor faces a dorsum side of ahand of a user. A dorsum area of the hand of the user is included as abackground area 420 in the depth image 410. The background area 420 maycorrespond to an area to be in contact with a touch object.

The touch gesture recognizing apparatus may obtain, using the depthsensor, the initial depth image based on a depth threshold value. Anarea having a depth value exceeding the depth threshold value in theinitial depth image may be set at a predetermined depth value. Aremaining area from which the background area 420 is excluded in thedepth image 410 may have a predetermined depth value based on the depththreshold value.

The touch gesture recognizing apparatus may extract, as a candidatebackground area, the background area 420 detected from the depth image410, and calculate a curvature of a boundary 430 of the background area420. For example, when a section in which the curvature of the boundary430 of the background area 420 is suddenly and significantly changedexists or an area longitudinally traversing the depth image 410 exists,the touch gesture recognizing apparatus may determine that thebackground area 420 is not a complete background area. When thecurvature of the boundary 430 of the background area 420 satisfies apredetermined learning condition, the touch gesture recognizingapparatus may learn a background area for detecting a touch input basedon the background area 420 extracted from the depth image 410.

FIG. 5 is a flowchart illustrating another example method of learning abackground area. A dorsum surface of a hand of a user, which correspondsto a background area facing a depth sensor of a watch type wearabledevice, may be changed based on a movement of a wrist of the user andthus, the background area facing the depth sensor may need to be learnedon a periodic basis. Subsequent to learning of a background area at aninitial state, a touch gesture recognizing apparatus may updateinformation on the learned background area by periodically learning abackground area to be in contact with a touch object.

Referring to FIG. 5, in operation 510, the touch gesture recognizingapparatus determines whether a predetermined learning cycle arrives.When the learning cycle is extremely short, a great system load mayarise and thus, the learning cycle may be determined based on aninfluence on a system load by an operation of learning a backgroundarea.

In operation 520, when the learning cycle is determined to arrive, thetouch gesture recognizing apparatus obtains a learning depth image usedfor learning a background area, and determines whether a touch event ispresent in the learning depth image. The touch gesture recognizingapparatus may determine whether the touch event, which indicates thatthe touch object is contacting a touch detection area, is present in thelearning depth image to determine whether a complete background areaexists.

When the touch event is present in the learning depth image in operation520, operation 530 is performed. In operation 530, the touch gesturerecognizing apparatus determines whether a size of the background areaincluded in the touch detection area satisfies a predetermined learningcondition. The size of the background area included in the touchdetection area may be changed due to a large movement of the user. Forexample, when the size of the background area included in the touchdetection area is greater than a predetermined threshold value, thetouch gesture recognizing apparatus may proceed to operation 540 tolearn the background area for an adjacent area that is adjacent to aboundary of the background area based on the learning depth image.Conversely, when the size of the background area included in the touchdetection area is less than or equal to the predetermined thresholdvalue, the touch gesture recognizing apparatus may return to operation510 to perform operations 510 and 520 as described in the foregoingdescription.

In operation 540, when the touch event is determined to be absent fromthe learning depth image in operation 520 or the size of the backgroundarea is determined to satisfy the predetermined learning condition inoperation 530 (e.g., the size of the background area included in thetouch detection area is greater than a predetermined threshold value),the touch gesture recognizing apparatus extracts a background area fromthe learning depth image. For example, the touch gesture recognizingapparatus may extract the background area from the learning depth imageusing an existing connected component method.

In operation 550, the touch gesture recognizing apparatus learns thebackground area extracted from the learning depth image. The touchgesture recognizing apparatus may learn the background area extractedfrom the learning depth image, and update information on a predeterminedbackground area based on a result of the learning.

FIG. 6 is a diagram illustrating another example operation of learning abackground area.

Referring to FIG. 6, a left box indicates a depth image including abackground area changed by a movement of a user. A touch detection area610 of the depth image includes a portion 630 of the changed backgroundarea and an adjacent area 620 that is adjacent to the portion 630. Theadjacent area 620 may indicate an area adjacent to the portion 630 in anarea having a depth value greater than a depth threshold value. Theportion 630 of the background area may correspond to a portion of adorsum area of a hand of the user. When the user moves a dorsum of thehand, a shape of the portion a 630 of the background area included inthe predetermined touch detection area 610 may be changed.

The touch gesture recognizing apparatus may update the background areaby learning a background area on a periodic basis to reduce an errorthat may be caused by a change in the background area. Referring to FIG.6, the right box indicates a result of learning and updating thebackground area. A range of a touch detection area 640 may differ from arange of the touch detection area 610 because the result of updating thebackground area is considered in the touch detection area 640.Subsequent to the learning and updating of the background area, thetouch detection area 640 may include a portion 660 of the updatedbackground area and an adjacent area 650 that is adjacent to the portion660. The touch gesture recognizing apparatus may recognize a touchgesture based on the updated touch detection area 640 until a nextlearning cycle arrives.

FIG. 7 is a flowchart illustrating an example method of detecting atouch input from a depth image. A touch gesture recognizing apparatusmay determine whether a touch input by a touch object exists based on atouch detection area including a portion of a background area includedin a depth image. For example, the touch gesture recognizing apparatusmay determine whether the touch object such as a finger makes contactwith a dorsum of a hand of a user based on the touch detection area. Thetouch gesture recognizing apparatus may determine a movement of thefinger of the user performed during the finger being in contact with thedorsum to be an intended motion of the user.

Referring to FIG. 7, in operation 710, the touch gesture recognizingapparatus determines a touch detection area. The touch gesturerecognizing apparatus may determine the touch detection area based on aboundary of a learned background area. The touch gesture recognizingapparatus may determine the touch detection area including a portion ofthe learned background area and an adjacent area that is adjacent to theportion based on the boundary of the learned background area.

In operation 720, the touch gesture recognizing apparatus analyzes abackground area included in the touch detection area. The touch gesturerecognizing apparatus may determine a number of autonomous adjacentareas detected in the touch detection area by performing a connectedcomponent image processing method on an adjacent area which is not abackground area in the determined touch detection area.

A touch input area formed by a touch object in the touch detection areamay be a complementary area to an adjacent area and thus, the touchgesture recognizing apparatus may determine information on the touchinput area using information on an autonomous adjacent area. Theinformation on the autonomous adjacent area may include, for example,information on a size or coordinates of a bounding box including theautonomous adjacent area, and information on the number of theautonomous adjacent areas. The information on the touch input area mayinclude, for example, information on a size or coordinates of a boundingbox including the touch input area, and information on the number oftouch input areas.

The information on the touch input area may be determined using theinformation on the autonomous adjacent area because reliability of depthinformation of the touch input area may be low due to a motion blur andnoise produced by a moving touch object. When the reliability of thedepth information of the touch input area is high, the touch gesturerecognizing apparatus may not determine the information on the touchinput area using the information on the autonomous adjacent area, butmay instead obtain the information on the touch input area by directlyperforming the connected component image processing method on the touchinput area.

In operation 730, the touch gesture recognizing apparatus determineswhether there is a contact of the touch object with a touch detectionarea based on a result of the analyzing performed in operation 720. Thetouch gesture recognizing apparatus may determine the contact of thetouch object and a number of touch inputs by the touch object based onthe number of the autonomous adjacent areas included in the touchdetection area. When the touch gesture recognizing apparatus determinesthat there is not contact of the touch object with the touch detectionarea, the number of the autonomous adjacent areas included in the touchdetection area may be one. Conversely, when the touch gesturerecognizing apparatus determines that there is contact of the touchobject with the touch detection area, the number of the autonomousadjacent areas included in the touch detection area may be at least two.For example, when the number of the autonomous adjacent areas includedin the touch detection area is determined to be two, the touch gesturerecognizing apparatus may determine that the touch object makes contactwith the background area.

In operation 740, when the touch object is determined to be in contactwith the background area, the touch gesture recognizing apparatusallocates an identifier to the touch input touched by the touch object.The touch gesture recognizing apparatus may allocate the identifier tothe touch input to track a motion of the touch object. When a pluralityof touch inputs are detected from the depth image based on the touchdetection area, the touch gesture recognizing apparatus may allocatedifferent identifiers to the touch inputs.

FIG. 8 is a diagram illustrating an example of an operation of detectinga touch input from a depth image.

In the example of FIG. 8, the upper left box indicates a depth imageobtained through a depth sensor, which includes a dorsum area of a handof a user as a background area 810. The upper right box includes abackground area 820 learned based on the background area 810. The middlebox includes a touch detection area 830 determined based on the learnedbackground area 820. A touch gesture recognizing apparatus may determinethe touch detection area 830 based on a boundary of the learnedbackground area 820. The touch detection area 830 includes a portion 850of the learned background area 820 and an adjacent area 840 that isadjacent to the portion 850. A number of autonomous adjacent areas maybe one because only the adjacent area 840 exists before a touch objectmakes contact.

The lower left box indicates a depth image obtained when the touchobject makes contact with a background area. The adjacent area 840included in the touch detection area 830 is divided into two autonomousadjacent areas, for example, an adjacent area 870 and an adjacent area880, by a touch input area 860 of the touch object. When a plurality ofautonomous adjacent areas is detected in the touch detection area, thetouch gesture recognizing apparatus may determine that a touch input mayoccur by the touch object.

When the touch input is determined to occur, the touch gesturerecognizing apparatus may determine information on the touch input area860 using information on the autonomous adjacent areas 870 and 880, orobtain the information on the touch input area 860 by performing aconnected component method on the touch input area 860. For example,when at least three autonomous adjacent areas are included in a touchdetection area, or a plurality of areas having different depth values ispresent in a touch input area, the touch gesture recognizing apparatusmay determine that a plurality of touch inputs is present in the touchdetection area.

FIG. 9 is a flowchart illustrating an example method of analyzing atouch input. As illustrated in FIG. 9, when a touch object is determinedto be in contact with a background area, a touch gesture recognizingapparatus may determine a location and a touch angle of a touch input byanalyzing the touch input.

Referring to FIG. 9, in operation 910, the touch gesture recognizingapparatus determines an x coordinate value of a touch input. The touchgesture recognizing apparatus may determine the x coordinate value ofthe touch input based on a touch input area included in a touchdetection area in a depth image. The touch gesture recognizing apparatusmay determine the x coordinate value of the touch input based on acoordinate axis in a width direction of the depth image. The touchgesture recognizing apparatus may convert a curved coordinate system ofthe background area to a planar coordinate system, and determine the xcoordinate value of the touch input in the planar coordinate system. Forexample, the touch gesture recognizing apparatus may calculate the xcoordinate value of the touch input based on Equation 1, as follows.

$\begin{matrix}{{TP}_{x} = \frac{B_{f\;{ma}\; x\;\_\; x} + B_{f\; m\; i\; n\;\_\; x}}{2}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

In Equation 1, “TP_(x)” denotes an x coordinate value of a touch inputon 2D coordinates.

“B_(f max) _(_) _(x)” and “B_(t min) _(_) _(x)” denote a maximum xcoordinate value and a minimum x coordinate value, respectively, of atouch input area included in a touch detection area. Based on Equation1, the touch gesture recognizing apparatus may determine, to be the xcoordinate value of the touch input, a mean value between the maximum xcoordinate value and the minimum x coordinate value of the touch inputarea detected in the touch detection area.

In operation 920, the touch gesture recognizing apparatus sets afiltering area in the touch input area. For example, the touch gesturerecognizing apparatus may set, to be the filtering area, an N×N sizearea based on the x coordinate value of the touch input determined basedon Equation 1. Here, “N” denotes a natural number indicating a number ofpixels.

In operation 930, the touch gesture recognizing apparatus sets afiltering area in an upper portion of a touch object area. The touchgesture recognizing apparatus may set the filtering area to which anoise filter is to be applied in the upper portion of the touch objectarea indicating an area of the touch object. For example, the touchgesture recognizing apparatus may determine a mean value between amaximum x coordinate value and a minimum x coordinate value of the upperportion of the touch object area to be an x coordinate value of theupper portion, and set an N×N size area based on the x coordinate valueof the upper portion to be the filtering area. Here, “N” denotes anumber of pixels, N being a natural number.

The touch object area may be an area in which the touch object appearsin the depth image and includes the touch input area. The touch inputarea may be a portion of the touch object area, and indicate a tip ofthe touch object to be in contact with the background area.

In operation 940, the touch gesture recognizing apparatus reduces noisein the filtering areas set in operations 920 and 930. A depth value ofthe touch input area to be output from the depth sensor may include afluctuation element and noise. When an original depth value of the touchinput area or the upper portion of the touch object area is applied, anerroneous z coordinate value may be obtained. The touch gesturerecognizing apparatus may reduce depth noise included in the touch inputarea or the upper portion of the touch object area by applying a noiseremoving filter to the touch input area or the upper portion of thetouch object area.

The touch gesture recognizing apparatus may apply an existing spatialnoise removing filter or temporal noise removing filter to the filteringareas set in operations 920 and 930 to reduce the noise included in thefiltering areas.

In operation 950, the touch gesture recognizing apparatus determines a zcoordinate value and a touch angle of the touch input. The touchgestures recognizing apparatus may determine the z coordinate value ofthe touch input on 2D coordinates using a depth value of the touch inputarea to which the noise removing filter is applied.

The touch gesture recognizing apparatus may determine the touch angle ofthe touch input using the depth value of the upper portion of the touchobject area to which the noise removing filter is applied. The touchgesture recognizing apparatus may determine the touch angle of the touchinput using 2D coordinate values of the touch input area, 2D coordinatevalues of the upper portion of the touch object area, and a heightdifference between the touch input area and the upper portion of thetouch object area.

The touch gesture recognizing apparatus may calculate a yaw angle and apitch angle of the touch input using coordinates P_(d) of the touchinput area, which is transformed to be three-dimensional (3D) Cartesiancoordinates, and coordinates P_(u) of the upper portion of the touchobject area, which is transformed to be 3D Cartesian coordinates.

For example, the touch gesture recognizing apparatus may calculate a 3Dtranslation element value to calculate the touch angle of the touchinput based on the following Equation 2, and calculate a 3D unit vectorto calculate the touch angle based on the following Equation 3. Thetouch gesture recognizing apparatus may calculate the yaw angle of thetouch input based on an element of the 3D unit vector as expressed inthe following Equation 4, and calculate the pitch angle of the touchinput based on an element of the 3D unit vector as expressed in thefollowing Equation 5.(T _(x) ,T _(y) ,T _(z))=(x _(p) _(u) −x _(p) _(d) ,y _(p) _(u) −y _(p)_(d) ,z _(p) _(u) −z _(p) _(d) )  [Equation 2]

In Equation 2, “T_(x),” “T_(y),” and “T_(z)” denote 3D translationelement values on x coordinates, y coordinates, and x coordinates,respectively. “x_(pu),” “y_(pu),” and “z_(pu)” denote locations of theupper portion of the touch object area on the x coordinates, the ycoordinates, and the z coordinates, respectively. “x_(pd),” “y_(pd),”and “z_(pd)” denote locations of the touch input area on the xcoordinates, the y coordinates, and the z coordinates, respectively.

$\begin{matrix}{\left( {u_{x},u_{y},u_{z}} \right) = \left( {\frac{T_{x}}{\sqrt{T_{x}^{2} + T_{y}^{2} + T_{z}^{2}}},\frac{T_{y}}{\sqrt{T_{x}^{2} + T_{y}^{2} + T_{z}^{2}}},\frac{T_{y}}{\sqrt{T_{x}^{2} + T_{y}^{2} + T_{z}^{2}}}} \right)} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack\end{matrix}$

In Equation 3, “u_(x),” “u_(y),” and “u_(z)” denote elements of the 3Dunit vector on the x coordinates, the y coordinates, and the zcoordinates, respectively. “T_(x),” “T_(y),” and “T_(z)” denote 3Dtranslation element values on three dimensions determined based onEquation 2.

$\begin{matrix}{{Deg}_{Yaw} = {\arccos\left( {- \frac{u_{y}}{\sqrt{1 - u_{z}^{2}}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

In Equation 4, “Deg_(Yaw)” denotes a yaw angle of a touch input, and“u_(y)” and “u_(z)” denote elements of the 3D unit vector on the ycoordinates and the z coordinates, respectively, determined based onEquation 3.Deg_(Pitch)=arccos(u _(z))  [Equation 5]

In Equation 5, “Deg_(Pitch)” denotes a pitch angle of a touch input, and“u_(z)” denotes an element of the 3D unit vector on the z coordinatesdetermined based on Equation 3.

The touch gesture recognizing apparatus may determine the touch angle ofthe touch input based on the yaw angle of the touch input calculatedbased on Equation 4 and the pitch angle of the touch input calculatedbased on Equation 5.

The touch gesture recognizing apparatus may determine the touch inputfrom the user by tracking a movement of the touch input on the 2Dcoordinates or tracking a change in the touch angle of the touch input.

When tracking a change in a location of the touch input, the touchgesture recognizing apparatus may use a method of alternating predictionand correction of a touch gesture in which the prediction and correctioncorrect each other based on Bayes' rule as expressed in the followingEquation 6. According to such a method, the touch gesture recognizingapparatus may improve a result of recognizing the touch gesture byrepeating operations of correcting the touch gesture based on apredicted value, predicting the touch gesture based on an input value towhich a weighted value is applied, and re-correcting an input value tobe subsequently input based on the predicted value.

$\begin{matrix}{{P\left( X \middle| z_{k} \right)} = \frac{{P\left( z_{k} \middle| X^{-} \right)}{P\left( X^{-} \right)}}{P\left( z_{k} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 6} \right\rbrack\end{matrix}$

In Equation 6, “X” denotes a process state, “k” denotes a step, and“Z_(k)” denotes an actual measurement value.

FIG. 10 is a diagram illustrating an example operation of analyzing atouch input.

Referring to FIG. 10, the upper left box illustrates a scene in which afinger 1020 of a user, which is a touch object, makes contact with adorsum area (background area) 1010 of a hand of the user. A depth sensorincluded in a touch gesture recognizing apparatus may generate a depthimage, which is illustrated as the upper right box and includes depthinformation on the scene illustrated in the upper left box. The depthimage may include information on a depth from the depth sensor to thetouch object.

The touch gesture recognizing apparatus may determine whether the touchobject makes contact with a background area based on a touch detectionarea. When the touch object is in contact with the background area, thetouch gesture recognizing apparatus may analyze a touch input area. Forexample, the touch gesture recognizing apparatus may determine an xcoordinate value of a touch input based on Equation 1 provided above.

The touch gesture recognizing apparatus may reduce noise included in thetouch input area before determining a z coordinate value of the touchinput based on a depth value. As illustrated in the middle left box, thetouch gesture recognizing apparatus may set a filtering area 1040 in thetouch input area included in a touch detection area 1030, and may reducenoise in the filtering area 1040 by applying a noise removing filter tothe filtering area 1040. Subsequently, the touch gesture recognizingapparatus may extract a depth value from the filtering area 1040, anddetermine the extracted depth value to be the z coordinate value of thetouch input.

As illustrated in the middle right portion of FIG. 10, the touch gesturerecognizing apparatus may determine a location 1050 of the touch inputon 2D coordinates based on the depth value extracted from the filteringarea 1040. An x coordinate value of the location 1050 of the touch inputmay indicate a location of the touch input based on a width direction ofthe depth image, and a z coordinate value of the location 1050 of thetouch input may indicate a depth value of the touch input.

Referring to the lower left box of FIG. 10, the touch gesturerecognizing apparatus may determine a touch angle of the touch inputusing a coordinate value and a depth value of an upper portion 1060 of atouch object area. As illustrated in a lower left box, using a methodsimilar to the method applied to the touch input area, the touch gesturerecognizing apparatus may set a filtering area 1070 in the upper portion1060 of the touch object area, and reduce noise in the filtering area1070 by applying a noise removing filter to the filtering area 1070.Subsequently, the touch gesture recognizing apparatus may extract adepth value from the filtering area 1070, and determine the touch angleof the touch input based on the depth value extracted from the filteringarea 1070.

FIG. 11 is a flowchart illustrating another example method ofrecognizing a touch gesture. The method of recognizing a touch gesturemay be performed by a touch gesture recognizing apparatus.

Referring to FIG. 11, in operation 1110, the touch gesture recognizingapparatus learns a background area based on a depth image. The touchgesture recognizing apparatus may learn the depth image for thebackground area in a state in which a touch object does not exist inorder to distinguish between the background area in the depth image anda touch object area in which the touch object appears in the depthimage.

The touch gesture recognizing apparatus may determine a background areato be used for a touch input by learning a background area included inan initial depth image, which is obtained in an initial state of anoperation. The touch gesture recognizing apparatus may determine whetherthe background area included in the initial depth image satisfies apredetermined learning condition. When the background area satisfies thelearning condition, the touch gesture recognizing apparatus may learnthe background area based on the initial depth image. The touch gesturerecognizing apparatus may extract the background area from the initialdepth image, and learn the background area to be in contact on the touchobject based on the extracted background area.

Subsequent to the initial state of the operation, the touch gesturerecognizing apparatus may continuously learn a background area includedin a depth image. Every time a predetermined learning cycle arrives, thetouch gesture recognizing apparatus may learn a background area from alearning depth image used for learning a background area, and update thebackground area to be in contact on the touch object based on a resultof the learning. The touch gesture recognizing apparatus may determine atouch detection area from the learned background area.

In operation 1120, the touch gesture recognizing apparatus obtains thedepth image in which the touch object and the background area arecaptured. The obtained depth image may include the touch object areaincluding depth information of the touch object.

In operation 1125, the touch gesture recognizing apparatus detects atouch input produced by the touch object in contact with the backgroundarea from the depth image, as set forth in detailed operations 1130 and1140. The detecting of the touch input is performed in a touch detectionarea based on the learned background area. In detailed operation 1130,the touch gesture recognizing apparatus determines the touch detectionarea based on the background area learned in operation 1110. The touchdetection area may include a portion of the learned background area andan adjacent area that is adjacent to a boundary of the learnedbackground area. For example, the background area may correspond to anentire area or a portion of a dorsum or a palm of a hand, or an arm areaof the user.

In detailed operation 1140, the touch gesture recognizing apparatusdetects the touch input applied by the touch object to the backgroundarea in the touch detection area. The touch gesture recognizingapparatus may determine whether the touch object makes contact with aboundary of the learned background area on the touch detection area. Thetouch gesture recognizing apparatus may determine whether the touchobject makes contact with a boundary of the learned background area onthe touch detection area based on a shape of the background areaincluded in the touch detection area. The touch gesture recognizingapparatus may detect at least one touch input within the touch detectionarea.

The touch gesture recognizing apparatus may detect a change in the touchinput while the touch object is in contact with the touch detectionarea. The touch gesture recognizing apparatus may determine a locationor a touch angle of the touch input. The touch gesture recognizingapparatus may determine an x coordinate value of the touch input basedon the touch object area in the depth image. The touch gesturerecognizing apparatus may reduce noise by applying a noise removingfilter to a filtering area determined based on the determined xcoordinate value, and determine a z coordinate value of the touch input.The touch gesture recognizing apparatus may determine the touch angle ofthe touch input based on an x coordinate value and a z coordinate valueof an upper portion of the touch object area.

In operation 1150, the touch gesture recognizing apparatus recognizes atouch gesture associated with the touch input by tracking a change inthe touch input. The touch gesture recognizing apparatus may recognizethe touch gesture associated with the touch input by tracking a changein the location or the touch angle of the touch input. The touch gesturerecognizing apparatus may determine a gesture pattern intended by theuser through the touch object by parsing the change in the location orthe touch angle of the touch input over time, and may generateinformation on the determined gesture pattern. When a plurality of touchinputs is detected, the touch gesture recognizing apparatus mayrecognize a touch gesture associated with the touch inputs based on amutual relation among the touch inputs.

FIG. 12 is a diagram illustrating an example configuration of a touchgesture recognizing apparatus 1200. Referring to FIG. 12, the touchgesture recognizing apparatus 1200 includes a depth sensor 1210, alearner 1220, and a processor 1230.

The touch gesture recognizing apparatus 1200 determines whether at leastone touch object such as a finger makes contact with a background areasuch as a dorsum area of a hand of a user using the depth sensor 1210such as a 3D depth camera. For example, when a plurality of touchobjects makes contact with the background area, the touch gesturerecognizing apparatus 1200 may track a motion of each touch object, andrecognize a gesture pattern corresponding to a touch input based on amutual relation among motions of the touch objects.

The touch gesture recognizing apparatus 1200 may generate a controlsignal corresponding to the motion of the touch object by recognizingthe motion of the touch object inputting the gesture pattern while thetouch object is in contact with the background area. For example,another device interworking with the touch gesture recognizing apparatus1200 may receive the control signal from the touch gesture recognizingapparatus 1200 and display a gesture pattern corresponding to an inputfrom the user.

The touch gesture recognizing apparatus 1200 may recognize the gesturepattern associated with the touch input by automatically detecting thetouch object in contact with the background area. The touch gesturerecognizing apparatus 1200 may recognize a more accurate gesture patternby obtaining location information of an area of the touch input througha depth image obtained by the depth sensor 1210.

The depth sensor 1210 may obtain the depth image. The depth sensor 1210may include an optical source (not shown) emitting light to an area, andan image sensor (not shown) receiving light reflected from the lightemitted from the optical source. For example, the optical source mayemit invisible light such as infrared light and ultraviolet light, orvisible light. The image sensor may include a pixel array configured toobtain distribution of the reflected light. The depth sensor 1210 maygenerate the depth image including information on a distance between theoptical source included in the depth sensor 1210 and the touch object,and an adjacent area using a time of flight (ToF) method.

The learner 1220 may learn a background area based on a learning depthimage obtained by the depth sensor 1210. For example, the learner 1220may learn, from the depth image, a background area corresponding to adorsum or a palm of a hand or an arm area of the user to which the touchinput applied by the touch object is to be input. The learner 1220 maylearn the depth image for the background area in a state in which thetouch object does not exist in the depth image in order to distinguishthe background area in the depth image from a touch object area in whichthe touch object appears in the depth image.

The learner 1220 may determine a background area to be used for thetouch input by learning a background area included in an initial depthimage obtained at an initial state of an operation. The learner 1220 mayextract the background area from the initial depth image, and learn thebackground area to be in contact on the touch object based on theextracted background area.

Subsequent to the initial state of the operation, the learner 1220 maycontinuously learn a background area included in the depth image. Ateach instance at which a predetermined learning cycle arrives, thelearner 1220 may learn the background area from the learning depth imageused for learning a background area, and update the background area tobe in contact on the touch object based on a result of the learning. Thelearner 1220 may determine whether a touch event is present on the touchdetection area based on learning a background area. When the touch eventis absent from the learning depth image, the learner 1220 may extract anadjacent area that is adjacent to a boundary of the background area fromthe learning depth image, and learn a new touch detection area based onthe extracted adjacent area.

The processor 1230 may determine a touch detection area based on thelearned background area, and detect a touch input applied by the touchobject within the touch detection area. The processor 1230 may determinewhether the touch object makes contact on the touch detection area, andtrack changes in the touch input while the touch object is in contact onthe touch detection area. The processor 1230 may recognize a touchgesture associated with the touch input by tracking a motion of thetouch input. The processor 1230 may recognize the touch gesturecorresponding to the motion of the touch input by tracking changes in alocation or a touch angle of the touch input.

The processor 1230 may determine a gesture pattern intended by the userthrough the touch object by parsing the changes in the location and thetouch angle of the touch input indicated in the depth image over time.The processor 1230 may then generate information on the determinedgesture pattern. The processor 1230 may determine a motion patternintended by the user using a learning-based method or an existinggesture recognizing method.

FIGS. 13A and 13B illustrate other example implementations of a touchgesture recognizing apparatus.

Referring to FIGS. 13A and 13B, a touch gesture recognizing apparatus1310 performs an interface controlling operation by interworking with anexternal device. For example, the touch gesture recognizing apparatus1310 may operate through interworking with a glass type wearable deviceas illustrated in FIG. 13A or with a display device illustrated in FIG.13B. In the example of FIG. 13A, a pair of smart glasses 1320 isillustrated as an example of the glass type wearable device. In theexample of FIG. 13B, a smart television (TV) 1330 is illustrated as anexample of the display device.

For example, when a user wears a watch type wearable device includingthe touch gesture recognizing apparatus 1310 and inputs a user input toa dorsum of a hand of the user using a finger or a pen, the touchgesture recognizing apparatus 1310 may recognize the user input made tothe dorsum and identify a gesture pattern corresponding to therecognized user input. The touch gesture recognizing apparatus 1310 maytransmit a control signal corresponding to the identified gesturepattern to an external device such as the smart glasses 1320 and thesmart TV 1330. The external device receiving the control signal from thetouch gesture recognizing apparatus 1310 may perform an interfaceoperation corresponding to the received control signal.

For example, when the user input is dragging in a certain direction onthe dorsum, the touch gesture recognizing apparatus 1310 may generate acontrol signal corresponding to the dragging and transmit the controlsignal to the external device. The external device receiving the controlsignal may, for example, perform an interface operation of moving acontent array displayed on the external device to a side or an interfaceoperation of selecting a next content from a current content displayedon the external device.

The apparatuses, units, modules, devices, and other componentsillustrated in FIGS. 1A-1C, 12, 13A and 13B that perform the operationsdescribed herein with respect to FIGS. 2, 3, 5, 7, 9 and 11 areimplemented by hardware components. Examples of hardware componentsinclude controllers, sensors, generators, drivers, displays and anyother electronic components known to one of ordinary skill in the art.In one example, the hardware components are implemented by one or moreprocessors or computers. A processor or computer is implemented by oneor more processing elements, such as an array of logic gates, acontroller and an arithmetic logic unit, a digital signal processor, amicrocomputer, a programmable logic controller, a field-programmablegate array, a programmable logic array, a microprocessor, or any otherdevice or combination of devices known to one of ordinary skill in theart that is capable of responding to and executing instructions in adefined manner to achieve a desired result. In one example, a processoror computer includes, or is connected to, one or more memories storinginstructions or software that are executed by the processor or computer.Hardware components implemented by a processor or computer executeinstructions or software, such as an operating system (OS) and one ormore software applications that run on the OS, to perform the operationsdescribed herein with respect to FIGS. 2, 3, 5, 7, 9 and 11. Thehardware components also access, manipulate, process, create, and storedata in response to execution of the instructions or software. Forsimplicity, the singular term “processor” or “computer” may be used inthe description of the examples described herein, but in other examplesmultiple processors or computers are used, or a processor or computerincludes multiple processing elements, or multiple types of processingelements, or both. In one example, a hardware component includesmultiple processors, and in another example, a hardware componentincludes a processor and a controller. A hardware component has any oneor more of different processing configurations, examples of whichinclude a single processor, independent processors, parallel processors,single-instruction single-data (SISD) multiprocessing,single-instruction multiple-data (SIMD) multiprocessing,multiple-instruction single-data (MISD) multiprocessing, andmultiple-instruction multiple-data (MIMD) multiprocessing.

The methods illustrated in FIGS. 2, 3, 5, 7, 9 and 11 that perform theoperations described herein with respect to FIGS. 1A-1C, 12, 13A and 13Bare performed by a processor or a computer as described above executinginstructions or software to perform the operations described herein.

Instructions or software to control a processor or computer to implementthe hardware components and perform the methods as described above arewritten as computer programs, code segments, instructions or anycombination thereof, for individually or collectively instructing orconfiguring the processor or computer to operate as a machine orspecial-purpose computer to perform the operations performed by thehardware components and the methods as described above. In one example,the instructions or software include machine code that is directlyexecuted by the processor or computer, such as machine code produced bya compiler. In another example, the instructions or software includehigher-level code that is executed by the processor or computer using aninterpreter. Programmers of ordinary skill in the art can readily writethe instructions or software based on the block diagrams and the flowcharts illustrated in the drawings and the corresponding descriptions inthe specification, which disclose algorithms for performing theoperations performed by the hardware components and the methods asdescribed above.

The instructions or software to control a processor or computer toimplement the hardware components and perform the methods as describedabove, and any associated data, data files, and data structures, arerecorded, stored, or fixed in or on one or more non-transitorycomputer-readable storage media. Examples of a non-transitorycomputer-readable storage medium include read-only memory (ROM),random-access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs,CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs,BD-Rs, BD-R LTHs, BD-REs, magnetic tapes, floppy disks, magneto-opticaldata storage devices, optical data storage devices, hard disks,solid-state disks, and any device known to one of ordinary skill in theart that is capable of storing the instructions or software and anyassociated data, data files, and data structures in a non-transitorymanner and providing the instructions or software and any associateddata, data files, and data structures to a processor or computer so thatthe processor or computer can execute the instructions. In one example,the instructions or software and any associated data, data files, anddata structures are distributed over network-coupled computer systems sothat the instructions and software and any associated data, data files,and data structures are stored, accessed, and executed in a distributedfashion by the processor or computer.

As a non-exhaustive example only, a wearable device described herein mayinclude a ring, a watch, a pair of glasses, a bracelet, an anklebracelet, a belt, a necklace, an earring, a headband, a helmet, or adevice embedded in clothing, a portable personal computer (PC) (such asa laptop, a notebook, a subnotebook, a netbook, or an ultra-mobile PC(UMPC), a tablet PC (tablet), a phablet, a personal digital assistant(PDA), a digital camera, a portable game console, an MP3 player, aportable/personal multimedia player (PMP), a handheld e-book, a globalpositioning system (GPS) navigation device, or a sensor, ahigh-definition television (HDTV), a DVD player, a Blu-ray player, orany other device capable of wireless or network communication. In oneexample, a wearable device is a device that is designed to be mountabledirectly on the body of the user, such as a pair of glasses or abracelet. In another example, a wearable device is any device that ismounted on the body of the user using an attaching device, such as asmart phone or a tablet attached to the arm of a user using an armband,or hung around the neck of the user using a lanyard.

While this disclosure includes specific examples, it will be apparent toone of ordinary skill in the art that various changes in form anddetails may be made in these examples without departing from the spiritand scope of the claims and their equivalents. The examples describedherein are to be considered in a descriptive sense only, and not forpurposes of limitation. Descriptions of features or aspects in eachexample are to be considered as being applicable to similar features oraspects in other examples. Suitable results may be achieved if thedescribed techniques are performed in a different order, and/or ifcomponents in a described system, architecture, device, or circuit arecombined in a different manner, and/or replaced or supplemented by othercomponents or their equivalents. Therefore, the scope of the disclosureis defined not by the detailed description, but by the claims and theirequivalents, and all variations within the scope of the claims and theirequivalents are to be construed as being included in the disclosure.

What is claimed is:
 1. A method of recognizing a touch gesture to beperformed by an apparatus for recognizing a touch gesture, the methodcomprising: obtaining, by a processor included in the apparatus, a depthimage in which a touch object and a background area are captured;detecting, by the processor, from the depth image, a touch input appliedby the touch object to the background area in a touch detection area;and recognizing, by the processor, a touch gesture associated with thetouch input by tracking a change in the touch input, wherein thebackground area is learned based on whether a touch event is present ina touch detection area of a learning depth image, wherein the touchevent is determined to be present in the touch detection area of thelearning depth image when a curvature of a background area satisfies apredetermined learning condition based on a change of the curvature, andwherein the curvature of the background area is determined to satisfythe predetermined learning condition when the change of the curvature ismore than a curvature threshold.
 2. The method of claim 1, furthercomprising: learning, by the processor, the background area; anddetermining, by the processor, the touch detection area from the learnedbackground area, wherein the detecting of the touch input is performedin the touch detection area based on the learned background area.
 3. Themethod of claim 2, wherein the learning of the background areacomprises: determining, by the processor, whether an initial backgroundarea on an initial depth image satisfies a predetermined learningcondition; and learning, by the processor, the background area based onthe initial depth image, in response to the background area satisfyingthe predetermined learning condition.
 4. The method of claim 2,comprising: determining, by the processor, whether a predeterminedlearning cycle arrives; obtaining, by the processor, the learning depthimage when the predetermined learning cycle arrives; determining, by theprocessor, when the predetermined learning cycle arrives, whether thetouch event is present in the touch detection area of the learning depthimage based on learning a background area for learning included in thelearning depth image; extracting, by the processor, an adjacent areathat is adjacent to a boundary of the background area for learning fromthe learning depth image, in response to the touch event being absent inthe touch detection area; and learning, by the processor, a new touchdetection area based on the extracted adjacent area.
 5. The method ofclaim 2, wherein the detecting of the touch input comprises:determining, by the processor, a touch detection area to detect thetouch input based on the learned background area; and detecting, by theprocessor, the touch input within the touch detection area.
 6. Themethod of claim 5, wherein the touch detection area comprises a portionof the learned background area and an adjacent area that is adjacent toa boundary of the learned background area.
 7. The method of claim 5,wherein the detecting of the touch input comprises: determining, by theprocessor, whether the touch object makes contact on the touch detectionarea; and detecting, by the processor, a change in the touch input, inresponse to the touch object being in contact on the touch detectionarea.
 8. The method of claim 7, wherein the determining of whether thetouch object makes contact on the touch detection area comprises:determining, by the processor, whether the touch object makes contactwith a boundary of the learned background area on the touch detectionarea based on a shape of the learned background area.
 9. The method ofclaim 5, wherein: the detecting of the touch input comprises detecting,by the processor, touch inputs within the touch detection area, and therecognizing of the touch gesture comprises recognizing, by theprocessor, a touch gesture associated with the touch inputs based on amutual relation between the detected touch inputs.
 10. The method ofclaim 1, wherein: the detecting of the touch input comprisesdetermining, by the processor, at least one of a location of the touchinput and a touch angle of the touch input; and the tracking of thechange in the touch input comprises tracking, by the processor, at leastone of a change in the location of the touch input and a change in thetouch angle of the touch input.
 11. A non-transitory computer-readablestorage medium storing instructions that, when executed by a processor,cause the processor to perform the method of claim
 1. 12. A method ofrecognizing a touch gesture to be performed by an apparatus forrecognizing a touch gesture, the method comprising: learning, by aprocessor, based on whether a touch event is present in a touchdetection area of a learning depth image, the background area to be incontact with a touch object; determining, by the processor, the touchdetection area based on the learned background area; detecting, by theprocessor, a touch input by the touch object based on a touch objectarea in the touch detection area; and recognizing, by the processor, atouch gesture corresponding to a motion of the touch input, wherein thetouch event is determined to be present in the touch detection area ofthe learning depth image when a curvature of a background area satisfiesa predetermined learning condition based on a change of the curvature,and wherein the curvature of the background area is determined tosatisfy the predetermined learning condition when the change of thecurvature is more than a curvature threshold.
 13. The method of claim12, wherein the learning of the background area to be in contact withthe touch object comprises: determining, by the processor, whether apredetermined learning cycle arrives; determining, by the processor,when the predetermined learning cycle arrives, whether the touch eventis present in the learning depth image; extracting, by the processor, abackground area for learning from the learning depth image, in responseto the touch event being absent in the learning depth image; andlearning, by the processor, the background area to be in contact withthe touch object based on the extracted background area.
 14. The methodof claim 12, wherein the detecting of the touch input comprises:determining, by the processor, whether the touch object makes contactwithin the touch detection area; and detecting, by the processor, achange in the touch input, in response to the touch object being incontact with the touch detection area.
 15. The method of claim 12,wherein: the detecting of the touch input comprises determining, by theprocessor, at least one of a location of the touch input and a touchangle of the touch input; and the recognizing of the touch gesturecomprises recognizing, by the processor, a touch gesture associated withthe touch input by tracking at least one of a change in the location ofthe touch input and a change in the touch angle of the touch input. 16.An apparatus for recognizing a touch gesture, comprising: a depth sensorconfigured to obtain a depth image in which a touch object is captured;and a processor configured to detect, from the depth image, a touchinput applied by the touch object in a background area and recognize atouch gesture associated with the touch input by tracking a change inthe touch input, wherein the background area is learned based on whethera touch event is present in a touch detection area of a learning depthimage, wherein the touch event is determined to be present in the touchdetection area of the learning depth image when a curvature of abackground area satisfies a predetermined learning condition based on achange of the curvature, and wherein the curvature of the backgroundarea is determined to satisfy the predetermined learning condition whenthe change of the curvature is more than a curvature threshold.
 17. Theapparatus of claim 16, wherein the processor is further configured to:learn a background area for an adjacent area that is adjacent to aboundary of the background area detected in the depth image; and updatethe touch detection area based on updating the learned background areaat predetermined learning cycles.
 18. The apparatus of claim 17, whereinthe processor is further configured to determine whether the touch eventis present in the touch detection area when a predetermined learningcycle arrives, and to extract a learning background area from thelearning depth image and learn the background area from the learningdepth image, in response to the touch event being absent in the touchdetection area.
 19. The apparatus of claim 17, wherein the processor isconfigured to determine the touch detection area for detecting the touchinput based on the learned background area, and detect a change in thetouch input within the touch detection area.
 20. The apparatus of claim16, wherein the apparatus is included in a wearable device.